Skip to content

query routing: multi-table plan options#4833

Merged
sougou merged 7 commits intovitessio:masterfrom
planetscale:ss-vrepl
Apr 25, 2019
Merged

query routing: multi-table plan options#4833
sougou merged 7 commits intovitessio:masterfrom
planetscale:ss-vrepl

Conversation

@sougou
Copy link
Copy Markdown
Contributor

@sougou sougou commented Apr 23, 2019

In this change the query routing takes the possibility that there could be multiple target options for a given table. The design for this is explained in #4790.

At a high level:

  • VSchema.FindTableOrVindex function can return a list of tables instead of a single one.
  • The route planbuilder creates multiple routeOptions, one for each table returned.
  • All actions that affected the plan of a route are changed to update all routeOptions.
  • If a particular routeOption cannot accommodate a pushed down construct, it's removed from the list. Previously, this was an error case. But if no options are left, then we return an error.
  • If two routeOptions qualify for a merge of routes, then all other combinations that don't qualify are discarded. This is the case for joins, subqueries and unions.

More details:

  • vindexTable was renamed to the more appropriate vschemaTable.
  • In order to achieve this, a new routeOption data type was introduced, and route was changed to contain a list of routeOptions.
  • In symtab, tables used to point at the vschema table that was used to build them. Since a table can now represent multiple target tables, this field has been moved into routeOption.
  • In symtab, columns used to contain a vindex member. Since this can change depending on the target table, the routeOption now contains a map of column to vindexes instead.
  • The routeOption also contains the vschemaTable. DMLs use this information. Since DMLs have to be more deterministic about the table they write to, they always choose the first option.
  • At the beginning of the Wireup phase, we evaluate all existing options and decide on the best available.

sougou added 6 commits April 22, 2019 20:00
@sougou
vschema: routing rules proto changes
0879118 
This is the first part of the changes to implement vitessio#4790.
This part implements all the management functionality for
routing rules.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
@sougou
vschema: routing rules in vindexes.VSchema
3f8848a 
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
@sougou
query routing: multi-table plan options
e0f79ce 
In this change the query routing takes the possibility that
there could be multiple target options for a given table. The
design for this is explained in vitessio#4790.

At a high level:
* VSchema.FindTableOrVindex function can return a list of
  tables instead of a single one.
* The route planbuilder creates multiple routeOptions, one
  for each table returned.
* All actions that affected the plan of a route are changed
  to update all routeOptions.
* If a particular routeOption cannot accommodate a pushed
  down construct, it's removed from the list. Previously,
  this was an error case. But if no options are left, then
  we return an error.
* If two routeOptions qualify for a merge of routes, then
  all other combinations that don't qualify are discarded.
  This is the case for joins, subqueries and unions.

More details:
vindexTable was renamed to the more appropriate vschemaTable.

In order to achieve this, a new routeOption data type was
introduced, and route was changed to contain a list of
routeOptions.

In symtab, tables used to point at the vschema table that
was used to build them. Since a table can now represent
multiple target tables, this field has been moved into
routeOption.

In symtab, columns used to contain a vindex member. Since
this can change depending on the target table, the routeOption
now contains a map of column to vindexes instead.

The routeOption also contains the vschemaTable. DMLs use
this information. Since DMLs have to be more deterministic
about the table they write to, they always choose the
first option.

At the beginning of the Wireup phase, we evaluate all existing
options and decide on the best available.

To be done:
When a table has multiple targets, the targets can have different
names than the original table. If so, the queries have to be
rewritten to address the new target tables. In order to do this,
each routeOption will contain a list of substitutions that will
be made during the Wireup phase.

Tests have to be written for the new flows.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
@sougou
query routing: add table substitutions
80bd0c2 
The code has also been refactored and cleaned up
for improved readability.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
@sougou
query routing: vschema and DML tests
3c45413 
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
@sougou
query routing: tests
11ed1a1 
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
@sougou sougou requested review from deepthi and rafael April 23, 2019 03:19
@sougou sougou mentioned this pull request Apr 24, 2019
Closed
deepthi
deepthi reviewed Apr 24, 2019
View reviewed changes
Copy link
Copy Markdown
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Some general questions:

  1. What is the behavior if no routing rules are specified?
  2. Can you specify routing rules for a subset of tables, and the rest would default to (1)?

go/vt/topo/helpers/compare_test.go
}

CopyRoutingRules(ctx, fromTS, toTS)
}
Copy link
Copy Markdown
Collaborator

@deepthi deepthi Apr 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should test here that routing rules are the same after Copy.

go/vt/vtctl/vtctl.go Outdated
"Applies the VTGate routing schema to the provided keyspace. Shows the result after application."},
{"GetRoutingRules", commandGetRoutingRules,
"",
"Displays the VSchema routing rules"},
Copy link
Copy Markdown
Collaborator

@deepthi deepthi Apr 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Displays the VSchema routing rules"},
"Displays the VSchema routing rules."},

go/vt/vtgate/planbuilder/builder.go Outdated
// result column and returns a distinct symbol for it.
SupplyCol(col *sqlparser.ColName) (rc *resultColumn, colnum int)

// Primitve returns the underlying primitive.
Copy link
Copy Markdown
Collaborator

@deepthi deepthi Apr 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Primitve returns the underlying primitive.
// Primitive returns the underlying primitive.

@sougou
vreplication: address review comments
7fc5662 
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
@sougou
Copy link
Copy Markdown
Contributor Author

sougou commented Apr 25, 2019

If no routing rules are specified, the existing rules apply. In other words, one has to specify routing rules only if they want to override the default behavior.

Basically, the routing rules are first applied. The output of that is used as input to the existing rules.

@sougou sougou merged commit eda3e15 into vitessio:master Apr 25, 2019
@setassociative setassociative mentioned this pull request Apr 26, 2019
Merged
setassociative pushed a commit to tinyspeck/vitess that referenced this pull request Apr 26, 2019
Merge upstream 2019-04-25 (#130)
3489b15 
It's been busy and exciting in vitess upstream. Some changes that felt worth calling out as they could impact assumptions about behavior:

- vitessio#4832 — this adds a timeout that is 3x the `healthCheckInterval` which at which point the tablet will stop reporting as healthy
- vitessio#4814 — this changes when command line args are used by vttablet
- vitessio#4811 — a planned reparent now sets `super_read_only`
- vitessio#4805 — behavior coming back from a failed backup is different

**Slack changes**  
        @demmer vitessio#4827 Correct suppress logging for begin...commit in autocommit
        @rafael vitessio#4824 adds timeouts for all statements
        @demmer vitessio#4826 adds logging of stack traces as opt-in
        @demmer vitessio#4819 remove begin/commit logs from autocommit txns
        @demmer vitessio#4796 improves support for vtexplain for begin/dml/dml/commit txns

**Non slack changes**  
        vitessio#4839 Improve behavior with reference table routing & vreplication
        vitessio#4833 Support query routing given there could be multiple targets for a table
        vitessio#4832 tablets get new health check behavior (health checks time out)
        vitessio#4837 refresh bug around row streamer handling creds
        vitessio#4830 apply a default value to the db_name attribute in local_metadata
        vitessio#4785 SHOW SCHEMAS aliased to SHOW DATABASES
        vitessio#4829 add experimental support for split clone & vertical split clone via vreplication
        vitessio#4822 do not normalize in order by
        vitessio#4791 Adds vreplication row streaming support
        vitessio#4814 command line flags now used regardless of management mode
        vitessio#4811 vttablet sets super_read_only during planned reparent
        vitessio#4803 xtrabackup testing
        vitessio#4727 support multiple vttablets running against one mysql instance
        vitessio#4746 TopoCat can now produce JSON output
        vitessio#4805 mysqld is now restarted after a failed backup
        vitessio#4685 macos bootstrap bug
        vitessio#4874 ZK opts and java land
        vitessio#4695 Adds support for xtrabackup
        vitessio#4794 fixes build failures
        vitessio#4725 changes mysql8.0 start args
        vitessio#4736 introduce new states to support vreplication
        vitessio#4788 expands orc error

**Docs etc**  
        vitessio#4831 lint
        vitessio#4827 docs
        vitessio#4816 const declaration cleanup
        vitessio#4820 const declaration cleanup
        vitessio#4825 docs
        vitessio#4818 docs
        vitessio#4809 docs
        vitessio#4812 moves consts around
        vitessio#4813 docs
        vitessio#4808 docs
        vitessio#4800 docs
        vitessio#4795 docs
@sougou sougou mentioned this pull request Aug 5, 2020
Merged
This was referenced Nov 30, 2022
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rafael rafael Awaiting requested review from rafael

1 more reviewer

@deepthi deepthi deepthi approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sougou @deepthi